3 Data Preparation

3 Data Preparation#

3.1 Data Selection Rationale#

Describe and justify the criteria and methods used to select the data from the initial dataset(s).
Discuss the relevance of the selected data to the data mining goals defined earlier.
Explain any assumptions made in this process and any implications for the final results.
Include the reasons for including or excluding certain attributes.

For the time-series we are going to search for and then select a full week of data.

Week 6 is complete.
Week 6 data stored in `df_selected_week` with length 672

_images/6ae8d29e0f160ce140eb8459771956d60bce5aa2ae196feb231f790cf69b0a01.png

3.2 Data Cleaning Report#

Detail the process of cleaning the data i.e. dealing missing values, duplicate records, or inconsistent entries.
Discuss the methods used for cleaning (e.g., deletion, imputation, etc.) and a justification of choices.
Highlight any challenges faced during the data cleaning process.
Provide before and after summaries to demonstrate the effects of our cleaning efforts.

3.3 Derived Attributes / Records#

Discuss any new attributes or records that were derived from existing ones, such as creating new features through binning, combining attributes, or performing calculations.
Explain why these transformations were done, and how these new attributes will contribute to the data mining goals.
If applicable, describe any generation of synthetic data or aggregation of data at different levels.

We are going to add a number of new features that are derived from existing data. These are periods for the strongest signals calculated using the Lomb-Scargle Periodogram. The purpose of this is to give context to the model for the position of values in the time series. This is especially important as there are a large number of missing days now that all non-complete days have been removed.

3.4 Merged Data#

Describe any data that was merged or joined together from different sources or tables.
Discuss the methods used to combine the data and any difficulties faced (like inconsistencies or discrepancies between the datasets).
Explain how any issues were resolved (e.g., how discrepancies were reconciled).
Detail how the merged data was validated to ensure correctness.

We are going to add the term dates to the data for use in the model. This will be a boolean value.

3.5 Reformatted Data#

Describe the process of reformatting the data for modeling or visualization purposes.
Discuss any transformations of the data into different types (e.g., numerical to categorical, date-time conversions), creation of dummy variables, or re-scaling of features.
Explain how and why any re-sampling procedures were done, if applicable.
Discuss the format that the final dataset(s) will be in for the subsequent modeling or visualization stage.

First we need to scale the data to improve GD (Gradient Descent) convergence and avoid vanishing/exploding gradients both of which slow down model training. We will use sklearn.preprocessing.MinMaxScaler as our data is discrete and follows a heavily right skewed mixture distribution with a long tail and an exponential component. It is important that we preserve the shape of original distribution.

Model 0 - Baseline#

This model takes the current value as input and returns it for the next time step, effectively predicting “no change”. As our model can currently only predict a single time step into the future and the prediction will not change as window length increases. We can add an option to increase the number of predictions that the model makes however and see how performance changes as the number of time steps is increased. For single time-steps the baseline should be a reasonable estimate as the number of pedestrians generally changes slowly throughout the day. First we need to change our pandas.core.series.Series object into a numpy.ndarray object. We are dealing with a single dimensional array and the values form a time vector i.e. they are an ordered list.

We then want to split the data into windows. In our first test run, we use a week of continuous data, and split it into windows of WINDOWSIZE=1 and HORIZON=1. The baseline model will take the input and return that value as the output. We will then calculate some scores to see how far the predictions are from the labels.

_images/4673bc78bca5dff4e5b407ddcaf8efa2096c1f8b607c9d5aa5f06be0bb246c9f.png

Input shape: (4, 1) | Target shape: (4, 1)
Inputs array shape: torch.Size([167, 4]) | Target array shape: torch.Size([167, 4])
Inputs type: <class 'torch.Tensor'> | Targets type: <class 'torch.Tensor'>

_images/3cf4134783ea8d272efe71608e248e5d33b283e55d763835be56aa3fb6fa1ffe.png

Model 0 Performance#

(torch.Size([668]), torch.Size([668]))

Mean Absolute Error (MAE): 0.05719786882400513
Mean Squared Error (MSE): 0.0077348146587610245
Root Mean Squared Error (RMSE): 0.0879477933049202
Symmetric Mean Absolute Percentage Error (SMAPE): 42.62365400791168%
R-squared (R²): 0.8907532938409718
      Model    MAE    MSE   RMSE   SMAPE    R^2
0  Baseline  0.057  0.008  0.088  42.624  0.891

Model 1: Simple LSTM#

For our first LSTM model, we will use a single LSTM layer and a linear output layer. The model is only capable of providing a single time step output, so the distance we want to predict into the future depends on the frequency of data points that we feed into out from our generated windows. To begin with we attempt to predict 15 minutes into the future as this is the highest frequency of measurements we have for training. Later on we may aggregate to hourly or daily measurements to allow us to predict further into the future.

Input shape: (40, 1) | Target shape: (1, 1)
Input shape: (40, 1) | Target shape: (1, 1)

Train input tensor: torch.Size([410, 40, 1]) | Train target tensor: torch.Size([410, 1, 1])

Test input tensor: torch.Size([182, 40, 1]) | Test target tensor: torch.Size([182, 1, 1])

Show code cell source Hide code cell source

from matplotlib import animation
from IPython.display import HTML
import matplotlib.pyplot as plt
import matplotlib


# A function to train the model and record the training data
def train_model_and_record_data(
    X_train,
    y_train,
    X_test,
    y_test,
    window_size=WINDOW_SIZE,
    epochs=EPOCHS,
    device=device,
):
    LSTM_v1 = LSTMModel1().to(device)
    optimiser = optim.Adam(LSTM_v1.parameters())
    loss_fn = nn.MSELoss()
    loader = data.DataLoader(
        data.TensorDataset(X_train, y_train.squeeze(-1)), shuffle=True, batch_size=8
    )

    training_data_per_window = []

    for epoch in range(epochs):
        LSTM_v1.train()
        for X_batch, y_batch in loader:
            y_pred = LSTM_v1(X_batch)
            loss = loss_fn(y_pred, y_batch)
            optimiser.zero_grad()
            loss.backward()
            optimiser.step()

        LSTM_v1.eval()
        with torch.inference_mode():
            y_pred = LSTM_v1(X_train)
            train_rmse = np.sqrt(loss_fn(y_pred, y_train.squeeze(-1))).item()
            test_pred = LSTM_v1(X_test)
            test_rmse = np.sqrt(loss_fn(test_pred, y_test.squeeze(-1))).item()

            train_plot = np.ones_like(timeseries) * np.nan
            y_pred = LSTM_v1(X_train)
            y_pred = y_pred[:, -1]
            train_plot[window_size:train_size] = np.expand_dims(
                y_pred.detach().numpy(), axis=1
            )

            test_plot = np.ones_like(timeseries) * np.nan
            test_pred = LSTM_v1(X_test)[:, -1]
            test_plot[train_size + window_size : len(timeseries)] = np.expand_dims(
                test_pred.detach().numpy(), axis=1
            )

            # Add the data for this epoch to the list
            training_data_per_window.append(
                (epoch, train_plot, test_plot, train_rmse, test_rmse)
            )

            if (epoch + 1) % 10 != 0:
                continue

            print(
                f"Window Size: {window_size} | Epoch: {epoch+1} | Train RMSE: {train_rmse:.4f} | Test RMSE: {test_rmse:.4f}"
            )

    return training_data_per_window

Input shape: (2, 1) | Target shape: (1, 1)
Input shape: (2, 1) | Target shape: (1, 1)
Input shape: (3, 1) | Target shape: (1, 1)
Input shape: (3, 1) | Target shape: (1, 1)
Input shape: (4, 1) | Target shape: (1, 1)
Input shape: (4, 1) | Target shape: (1, 1)
Input shape: (6, 1) | Target shape: (1, 1)
Input shape: (6, 1) | Target shape: (1, 1)
Input shape: (8, 1) | Target shape: (1, 1)
Input shape: (8, 1) | Target shape: (1, 1)
Input shape: (10, 1) | Target shape: (1, 1)
Input shape: (10, 1) | Target shape: (1, 1)
Input shape: (15, 1) | Target shape: (1, 1)
Input shape: (15, 1) | Target shape: (1, 1)
Input shape: (20, 1) | Target shape: (1, 1)
Input shape: (20, 1) | Target shape: (1, 1)

(torch.Size([209, 10, 1]), torch.Size([209, 1, 1]), torch.Size([209, 4, 1]))

_images/886f13da5cc88062374661864a9756669ec0da6f10d7e5bce0b1647450bcdaa7.png

Windowing, also known as “rolling windows” or “sliding windows,” is a common method used in time series analysis and is often used with machine learning models. However, whether it’s essential or not depends largely on the specific task, data characteristics, and the model being used.

Here are some points to consider:

Sequence Learning: Many machine learning models, especially those used for time series like Recurrent Neural Networks (RNNs) and its variants (like LSTM and GRU), require input in the form of sequences. They make predictions based on the “memory” of past data points. Windowing is a way of creating these sequences from a time series.

Stationarity: Windowing can help make a time series more stationary (i.e., its properties do not depend on the time at which the series is observed) by creating a series of differences between values in the window, which is a requirement for certain models.

Feature Engineering: Windowing can also be used for feature engineering, where you create new features based on a window of values - such as rolling averages, standard deviations, etc. This can help capture trends and seasonality in the data which can be crucial for the performance of the models.

Reduce Noise: A windowed average can sometimes help reduce noise and smooth out short-term fluctuations, which may be beneficial for the model.

Temporal Dependencies: In time-series problems, current outputs can be dependent on previous inputs or outputs (auto-regressive nature). Windowing allows capturing these temporal dependencies.

However, it’s worth noting that windowing is just one of many techniques in time series analysis. Depending on the nature of your problem, other techniques may also be applicable and useful. It is always recommended to understand your data and problem thoroughly and choose the techniques that are most appropriate for your specific use case.

Aside from windowing, there are various techniques used in time series analysis, including:

Autoregression (AR): Models the next step in the sequence as a linear function of the observations at prior time steps.
Moving Average (MA): Models the next step in the sequence as a linear function of the residual errors from a mean process at prior time steps.
Autoregressive Moving Average (ARMA): Models the next step in the sequence as a linear function of the observations and residual errors at prior time steps.
Autoregressive Integrated Moving Average (ARIMA): An extension of ARMA that can model a wide range of time series data.
Seasonal Autoregressive Integrated Moving-Average (SARIMA): An extension of ARIMA that explicitly supports univariate time series data with a seasonal component.
Vector Autoregression (VAR): Models the next step in each time series using an AR model.
Vector Autoregression Moving-Average (VARMA): Models the next step in each time series using an ARMA model.
Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX): An extension of the VARMA model that also includes the modeling of exogenous variables.
Simple Exponential Smoothing (SES): Models the next time step as an exponentially weighted linear function of observations at prior time steps.
Holt Winter’s Exponential Smoothing (HWES): Models the next step in the sequence as an exponentially weighted linear function of observations at prior time steps, taking trends and seasonality into account.